The invention relates generally to image-based rendering, and more specifically to using manifold mosaics to accomplish image-base rendering.
Image-based rendering (IBR) in general simulates a continuous range of virtual camera viewpoints of a scene or object from a discrete set of input images of the scene or object. IBR can be used for a variety of different applications. For example, a currently popular application is to use IBR in conjunction with electronic commerce-type applications on the Internet. A user interested in seeing a car""s interior or exterior from different viewpoints, for example, may be able to rotate inside a virtual camera inside the car""s interior, and zoom in and out to see more or less of the interior, and may also be able to rotate a virtual camera around the car""s exterior. Thus, using IBR in such instances enables consumers to better see physical real-world objects that they may desire to purchase, allowing those consumers to make better-informed decisions.
Within the prior art, IBR has been accomplished in large part by using what is know as view interpolation. View interpolation generally involves using a number of known views to generate a desired view, using interpolation of the known views. View interpolation is, however, a difficult task to accomplish. For example, feature correspondencexe2x80x94the corresponding of features, such as pixels or objects, within one of the known views to one or more of the other of the known viewsxe2x80x94usually needs to be accurately established. Correspondence is particularly difficult when the two known views, or images, have been taken from distant viewpoints.
However, when accurate depth information of a to-be-rendered image is availablexe2x80x94that is, information regarding the depth of the desired view relative to a subject object or scenexe2x80x94the image can be rendered from a close known view by accomplishing what is known as warping. That is, the close known view is modified according to its depth. However, providing precise depth information regarding a to-be-rendered image is generally difficult to accomplish.
Many prior art IBR techniques solve the difficulties of correspondence and precise depth information by using a substantial number of images. To render an image at a new viewpoint, these techniques generally select nearby rays of known images, and interpolate them. Constant depth assumptions are made about the depth of the scene in order to locate optimal close rays, for optimal rendering quality. However, the large number of images needed for these techniques means that a large amount of data is required, which is undesirable for Internet and other applications, where the bandwidth over which the images will be transmitted may be low, resulting in inconveniently long transmission times, and/or where the storage space available for the images may be relatively small, such as in specialized devices like Internet appliances that may not have large hard disk drives or similar types of storage devices.
To solve this problem, the prior art typically compromises the extent to which the subject scene can be viewed by the user by constraining the motion of the virtual rendering camera. For example, some systems only allow users to explore large environments at pre-specified locations. However, even though a continuous change in viewing directions at each location is allowed, such systems can only jump among the pre-specified locations when a different viewing location is desired. This causes visual discontinuity and discomfort for the user. Thus, while the amount of data necessary for such IBR techniques can be reduced, the compromise is that perceived continuous virtual camera movement can be lost. Unfortunately, perceived continuous virtual camera movement is generally considered to be important for a user to smoothly navigated within a virtual environment.
For this and other reasons, therefore, there is a need for the present invention.
The invention relates to manifold mosaic hopping for image-base rendering (IBR). In one embodiment, a set of manifold mosaics, such as concentric mosaics, is constructed. Each mosaic represents a number of unique viewpoints that different from one another in only one direction. For example, in the case of concentric mosaics, the viewpoints in a given mosaic differ from one another only in the angular direction, in the r-theta domain. From a current virtual camera or viewing position within the set of mosaics, a new position can be moved to, such that the movement is continuous over any one mosaic, and is discrete across the mosaics. For example, in the case of concentric mosaics, movement is continuous over a given mosaic""s circle, and is discrete when moving in the radial direction across mosaics. An image is rendered by local warping, without interpolation of more than one mosaic.
Embodiments of the invention provide for a reduction in the amount of dataxe2x80x94that is, the number of imagesxe2x80x94needed to render a new image based on a new viewpoint of a scene or object, without unduly constraining the locations at which the new image can be rendered. The number and location of the manifold mosaics can be selected such that the discrete movement across mosaics perceptually appears to the user as continuous movement. That is, the number and location of the mosaics can be selected to take advantage of insensitive characteristics of the human eye, such that perceived smooth movement is still achieved even though discrete movement is in fact taking place.